Audio recognition-based industrial automation control

ABSTRACT

A system for performing industrial automation control may include an audio device that receives audio data from an element in an industrial automation system. The audio device may determine orientation data based on the audio data. In addition, the audio device may determine an automation command to control a machine in the industrial automation system based on the audio data and the orientation data. After determining the automation command, the audio device may implement a first control action for the machine based at least in part on the automation command, where the first control action causes the machine to adjust an operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/116,605, filed Aug. 29, 2018, entitled “AUDIO RECOGNITION-BASEDINDUSTRIAL AUTOMATION CONTROL,” which will issue as U.S. Pat. No.10,719,066 on Jul. 21, 2020, which is incorporated by reference for allpurposes.

BACKGROUND

The disclosure relates generally to systems and methods for controllingdevices within an industrial automation system. More particularly,embodiments of the present disclosure are related to systems and methodsfor controlling devices in the industrial automation system using audiodevices disposed within the industrial automation system to improverobustness, reliability, and accuracy of operations within industrialenvironment.

Industrial automation systems are managed and operated using automationcontrol and monitoring systems, particularly in industrial automationenvironments. Such applications may include the powering of a wide rangeof actuators, such as valves, electric motors, and so forth, and thecollection of data via sensors. Typical automation control andmonitoring systems may include one or more components, such asprogramming terminals, automation controllers, input/output (I/O)modules, communication networks, human-machine interface (HMI)terminals, and the like.

Generally, certain precautions are taken to ensure that devices inindustrial automation systems are operated as desired. However, someindustrial automation systems are limited in controlling its devicesusing the automation control and monitoring systems described above.Although these automation control and monitoring systems may be used tomanage the operations of the devices within the industrial automationsystem, improved systems and methods for operating devices within anindustrial automation system are desirable.

BRIEF DESCRIPTION

In one embodiment, a method for performing industrial automation controlin an industrial automation system is described. The method may includedetecting, via an audio device, audio data by a user in the industrialautomation system. The method may also include determining, via theaudio device, orientation data associated with the user based at leastin part on the audio data. In addition, the method may also includedetermining, via the audio device, an automation command to control amachine based at least in part on the audio data and the orientationdata and implementing, via the audio device, a first control action forthe machine based at least in part on the automation command.

In another embodiment a system for performing industrial automationcontrol is described. The system may include an audio device thatreceives audio data of an element in an industrial automation system.The audio device may determine an orientation of an element based on theaudio data and an automation command for controlling a machine in theindustrial automation system based on the audio data, the orientation ofthe element, automation commands stored in a database, and audiodatasets associated with the stored automation commands stored in thedatabase. In addition, the audio device may send a first control actionto the machine based at least in part on the automation command. Thefirst control action may cause the machine to adjust an operation.

In yet another embodiment, a tangible, non-transitory computer-readablemedium that stores instructions executable by a processor of anelectronic device that may cause the processor to detect audio dataassociated with a first machine or user in an industrial automationsystem. The processor may also determine orientation data associatedwith the first machine or user based on the audio data. Afterdetermining orientation data, the processor may determine an automationcommand to control a second machine in the industrial automation systembased on the audio data and the orientation data. In addition, theprocessor may also send a first control action to the second machinebased at least in part on the automation command, wherein the firstcontrol action causes the second machine to adjust an operation.

DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure may become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a perspective view of an example industrial automation system,in accordance with an embodiment;

FIG. 2 is a block diagram of an example audio device of the industrialautomation system of FIG. 1, in accordance with an embodiment;

FIG. 3 is a block diagram of multiple audio devices of FIG. 2, inaccordance with an embodiment;

FIG. 4 is a machine and multiple audio devices of the industrialautomation system of FIG. 1, in accordance with an embodiment;

FIG. 5 is a block diagram of a control system that operates theindustrial automation system of FIG. 1, in accordance with anembodiment;

FIG. 6 is a flowchart of a method for controlling one or more deviceswithin the industrial automation system of FIG. 1 using the audio deviceof FIG. 2, in accordance with an embodiment;

FIG. 7 is a flowchart of a method for monitoring one or more deviceswithin the industrial automation system of FIG. 1 using the audio deviceof FIG. 2, in accordance with an embodiment; and

FIG. 8 is a flowchart of a method for controlling one or more deviceswithin the industrial automation system of FIG. 1 based on whetherhumans or objects are present as detected by the audio device of FIG. 2,in accordance with an embodiment.

DETAILED DESCRIPTION

One or more specific embodiments of the present disclosure will bedescribed below. In an effort to provide a concise description of theseembodiments, all features of an actual implementation may not bedescribed in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements. One ormore specific embodiments of the present embodiments described hereinwill be described below. In an effort to provide a concise descriptionof these embodiments, all features of an actual implementation may notbe described in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

The present disclosure is generally directed towards an audio devicedisposed in an industrial automation system that may detect sounds oraudio spoken or emitted by persons or objects within a spaceencompassing the industrial automation system. After detecting theaudio, the audio device may perform various operations in the industrialautomation system based on the detected audio. That is, the audio devicemay detect audio produced by persons and/or objects (e.g., machines,motors, robots) within the industrial automation system space and usethe detected audio to control various industrial automation deviceswithin the industrial automation system. In addition to detecting audio,the audio device, in some embodiments, may detect emotions to determinean urgency of the detected audio and/or a presence of elements in theindustrial automation system, as well as orientations of the detectedelement (e.g., persons and/or objects) in the industrial automationsystem. The audio device may perform various operations in theindustrial automation system based on the detected element and/or theorientation of the detected element.

By using the audio device in the industrial automation system to detectaudio of persons and/or objects, an operator may control operations ofthe industrial automation system while being located physically awayfrom an object that the command is directed towards. For example, aperson may be loading boxes onto a conveyor belt, which may transportthe boxes to a second machine that wraps the boxes. While the secondmachine is in the line of sight of the person, the person may verballyrequest that the second machine stops its operation by directing acommand to an audio device disposed on the second machine. In this way,the second machine, which may be inaccessible to the person while he/sheis loading the boxes, may be controlled by the person without the personphysically interacting with the second machine.

With this in mind, the presently disclosed embodiments include an audiodevice that may be used in various industrial automation processesincluding remotely controlling machines associated with the industrialautomation system, remotely executing safety procedures to improvework-place safety for personnel, and the like. In one embodiment, aperson located within the industrial automation system space mayremotely change the behavior of an object or machine of the industrialautomation system based at least in part on verbal commands. That is,persons located within the industrial automation system space may directquestions to an audio device that may be associated with a particularcomponent in the industrial automation system to learn more about thecomponent. For example, a person may inquire about informationconcerning a flow rate of a pipe, a load voltage of a particular motorin the industrial automation system, a purpose or function of aparticular object, or any other suitable question. In response to theinquiry, the audio device may interpret the information request, query adatabase or component to determine an answer for the informationrequest, and audibly respond to the question.

In some embodiments, an industrial environment may be equipped withmultiple audio devices that control the operations of multiplecomponents in the environment. To ensure that detected commands aredirected to the appropriate audio device, each audio device may includea number of microphones that detect a directionality or orientation ofthe person providing the audible command. Based on the strength of theaudio signal detected by each microphone in one or more audio devices,the directionality of the audible command may be determined, and thusthe target audio device may also be identified. The targeted audiodevice may then perform the audible command.

In some embodiments, the audio device may also operate as an independentunit in the industrial automation system and detect directionality of acommand based on an array of microphones and the corresponding strengthof signals. Alternatively, each audio device may be designed to detect alimited set of commands that are directed to a component assigned to therespective audio device. In addition, multiple audio devices may operateas a collective unit in the industrial automation system. As acollective unit, the multiple audio devices may each receive audioinputs and, through the collective analysis of the multiple audio inputsand/or through communication with a control system, the control systemmay output information and/or control operations as requested. Forexample, the multiple audio devices may use triangulation schemes todetermine directionality of a verbal command or the object at which theverbal command is directed towards, to determine a source location of anunexpected sound or where the unexpected sound is emitted from, and thelike. Additional details regarding the audio device and various systemsand methods for coordinating operations between the audio device andcomponents of the industrial automation system are described in moredetail with reference to FIGS. 1-8.

By way of introduction, FIG. 1 is a perspective view of an exampleindustrial automation system 10 controlled by an industrial controlsystem 11. The industrial automation system 10 includes stations havingmachine components and/or machines to conduct a particular functionwithin an automated process, for example, a beverage packaging process,as is depicted. The automated process may begin at a station 13 used forloading objects, such as empty cans or bottles to be filled, into theindustrial automation system 10 via a conveyor section 14. The conveyorsection 14 may transport the objects to a station 16 to perform a firstaction, for example, washing the empty cans and/or bottles. As objectsexit from the station 16, the conveyor section 14 may transport theobjects to a station 20, such as a filling and sealing station, in asingle-file line. A second conveyor section 14 may transport objectsfrom the station 20 to a station 26. After the objects proceed throughthe various stations, the objects may be removed from the station 28,for example, for storage in a warehouse 30. Clearly, for otherapplications, the particular system, machine components, machines,stations, and/or conveyors may be different or specially adapted to theapplication.

For example, the industrial automation system 10 may include machineryto perform various operations in a compressor station, an oil refinery,a batch operation for making food items, a mechanized assembly line, andso forth. Accordingly, the industrial automation system 10 may comprisea variety of operational components, such as electric motors, valves,actuators, temperature elements, pressure sensors, or a myriad ofmachinery or devices used for manufacturing, processing, materialhandling, and other applications.

Additionally, the industrial automation system 10 may include varioustypes of equipment that may perform the various operations that may bepart of an industrial application. For instance, industrial automationsystem 10 may include electrical equipment, hydraulic equipment,compressed air equipment, steam equipment, mechanical tools, protectiveequipment, refrigeration equipment, power lines, hydraulic lines, steamlines, and the like. Some example types of equipment may include mixers,machine conveyors, tanks, skids, specialized original equipmentmanufacturer machines, and the like. In addition to the equipmentdescribed above, the industrial automation system 10 may also includemotors, protection devices, switchgear, compressors, and the like.

In certain embodiments, one or more properties of the industrialautomation equipment, such as the station 16, may be monitored andcontrolled by an industrial control system 11 for regulating controlvariables. For example, sensing devices (e.g., sensors 31) may monitorvarious properties of the industrial automation system 10 and may beused in adjusting operations of the industrial automation system 10. Insome cases, the industrial automation system 10 may be associated withdevices used by other equipment. For instance, scanners, gauges, valves,flow meters, and the like may be disposed on the industrial automationsystem 10. Here, the industrial control system 11 may receive data fromthe associated devices and use the data to perform their respectiveoperations more efficiently. For example, a controller of the industrialautomation system 10 associated with a motor drive may receive dataregarding a temperature of a connected motor and may adjust operationsof the motor drive based on the data.

The industrial control system 11 may be communicatively coupled to adisplay/operator interface 32 (e.g., a HMI) and to devices of theindustrial automation system 10. Although one industrial control system11 is depicted, it should be understood that any suitable number ofindustrial control systems 11 may be used in a particular industrialautomation system 10 embodiment. The industrial control system 11 mayfacilitate representing components of the industrial automation system10 through programming objects that may be instantiated and executed toprovide simulated functionality similar or identical to the actualcomponents, as well as visualization of the components, or both, on thedisplay/operator interface 32. The programming objects may include codeand/or instructions stored in the industrial control system 11 andexecuted by processing circuitry of the industrial control system 11.The processing circuitry may communicate with memory circuitry to permitthe storage of the component visualizations.

As illustrated, the display/operator interface 32 depictsrepresentations 33 of the components of the industrial automation system10. The industrial control system 11 may use data transmitted by sensors31 to update visualizations of the components via changing one or morestatuses, states, and/or indications of current operations of thecomponents. These sensors 31 may be any device adapted to provideinformation regarding process conditions. Indeed, the sensors 31 may beused in a process loop that are monitored and controlled by theindustrial control system 11. As such, a process loop may be activatedbased on process inputs (e.g., an input from the sensor 31) or directinput from a person 38 via the display/operator interface 32. The person38 operating and/or monitoring the industrial automation system 10 mayreference the display/operator interface 32 to determine variousstatuses, state, and/or current operations of the industrial automationsystem 10 and/or for a particular component. Furthermore, the personoperating and/or monitoring the industrial automation system 10 mayadjust to various components to start, stop, power-down, power-on, orotherwise adjust an operation of one or more components of theindustrial automation system 10 through interactions with control panelsor various input devices.

A person 38 controlling or monitoring operation of the industrialautomation system 10 may find controlling or monitoring difficult ifthey walk away or are otherwise located away from the display/operatorinterface 32 or from a respective control panel. Thus, an audio device40 may assist the person 38 in remotely monitoring components of theindustrial automation system 10 (e.g., while located away from thedisplay/operator interface 32) via verbal or audible commands. The audiodevice 40 may perform actions in response to verbal or audible commandsof the person 38. For instance, the audio device 40 may interpret theaudible commands, determine a corresponding command for an associatedcomponent, and adjust the operation of the component based on thecommand. In this way, the person 38 may physically input a change to theoperation of a first machine via an interface of the first machine,while using the audio device 40 to change operation of a second machineusing verbal or audible commands without physically accessing secondmachine. For example, the person 38, while at the station 26 mayverbally request the audio device 40 to report on or change theoperation of the station 28 while physically located away from thedisplay/operator interface 32 or control panel for the station 28. As anadditional example, if a person were to fall into a cavity or get lockedinto a secure space, the person may speak a verbal command or triggerword/phrase to instruct the audio device to enable a light component(e.g., to strobe the light, to blink, to turn to a solid color of light,to change color of light emitted) or some other annunciation output toindicate his/her presence within the area to other nearby personnel. Inthis way, the presently disclosed embodiments may provide a way toverbally generate and transmit a signal that indicates a presence withinthe area to other personnel.

Based on a signal strength of the audible command, the audio devices 40may determine an orientation of the person 38 and a direction of theaudible command to facilitate selection of a machine that the audiblecommand is directed towards. In some embodiments, the audio device 40may detect the audible command projected from a person and confirm thatthe person has a line of sight with respect to the component beingcontrolled before implementing the associated command. Examples ofdevices the audio device 40 may control or change operation of include,but are not limited to, actuators, electric motors, electric drives,electric presses, and the like.

To help elaborate, FIG. 2 is a block diagram representation of an audiodevice 40 including an array of multiple microphones 42. It should beunderstood that any suitable sound-receiving device may be used in placeof, or in addition to, the microphones 42, for example, a combinedspeaker and microphone device, or a singular microphone 42 may beincorporated into the audio device 40. The audio device 40 may alsoinclude processing circuitry 44 including a communication component 46,a processor 48, a memory 50, I/O ports 52, and the like. Thecommunication component 46 may be a wireless or a wired communicationcomponent that may facilitate communication between the audio device 40and other audio devices 40, machines, and/or the industrial controlsystem 11 via a communication network 54. This wired or wirelesscommunication protocols may include any suitable communication protocolinclude Wi-Fi, mobile telecommunications technology (e.g., 2G, 3G, 4G,LTE), Bluetooth®, near-field communications technology, and the like.The communication component 46 may include a network interface to enablecommunication via various protocols such as EtherNet/IP®, ControlNet®,DeviceNet®, or any other industrial communication network protocol.

The processor 48 of the audio device 40 may be any suitable type ofcomputer processor or microprocessor capable of executingcomputer-executable code, including but not limited to one or more fieldprogrammable gate arrays (FPGA), application-specific integratedcircuits (ASIC), programmable logic devices (PLD), programmable logicarrays (PLA), and the like. The processor 48 may, in some embodiments,include multiple processors. The memory 50 may include any suitablearticles of manufacture that serve as media to storeprocessor-executable code, data, and the like. The memory 50 may storeprocessor-executable code used by the processor 48 to perform thepresently disclosed techniques.

Generally, the audio device 40 may receive (e.g., detect) audio datarelated to the person 38, a machine, and/or any nearby object or theambient environment via one or more audio sensors (e.g., microphones 42)communicatively coupled to one or more of the I/O ports 52. Uponreceiving audio data, the audio device 40, via the processor 48, mayinterpret the audio data to determine automation commands or actions forthe audio device 40 to perform in response to the determined automationcommand. In some embodiments, that the determined automation command maybe forwarded to interpret the detected audio data, the audio device 40may analyze characteristics of the audio data, including frequency(e.g., pitch), amplitude (e.g., loudness) of the audio data, or anysuitable characteristic used to distinguish one verbal command fromanother verbal command, to determine if the audio data matches thecharacteristics of an audible automation command stored, learned, orotherwise interpretable by the audio device 40. If a threshold or morecharacteristics match, the audio device 40 determines an automationcommand based on the audio data.

In this way, the audio device 40 may be a stand-alone componentconnected through a direct input/output coupling to the industrialcontrol system 11, components and/or machines of the industrialautomation system 10, and the like. The audio device 40 may work with asystem of audio devices 40 networked with the industrial control system11. Each audio device 40 in a networked system may have a uniqueidentifier and certain data disseminated from the audio device 40 may betagged with the unique identifier. As discussed above, the audio device40 may be communicatively coupled to the communication network 54, whichmay include an Internet connection, or otherwise suitable wireless orwired communicative coupling to expand its interpretation and functionalcapabilities, but, in some embodiments, the audio device 40 may not relyon such a communicative coupling. In other words, the audio device 40may have particular capabilities that may function without an Internet,wireless, or wired connection. For example, the audio device 40 mayperform local command interpretation without an Internet or wirelessconnection.

In addition, the audio device 40 may use detected audio strength data(e.g., as visualized in graph 58) corresponding to a directionality ofthe audio data from the person 38 to facilitate determining a context toan automation command (e.g., which machine or component the command isdirected towards). For example, the audio device 40 may determine whichmicrophones 42 are receiving the strongest audio data, and in this waythe audio device 40 may determine a directionality of the audiblecommand. For example, as shown if FIG. 2, microphone 42A has a largerdetected audio strength (e.g., depicted in graph 58A) than themicrophone 42B with the detected audio strength data (e.g., depicted ingraph 58B). Also, the audio device 40 may use secondary input data fromthe sensors 31, the industrial control system 11, or the like toincrease confidence in a determined automation command from the audiodata, verify a determined automation command, determine whether theautomation command should be implemented, or the like. When noise ispresent in one area, the processor may use another microphone in anotherdirection to confirm command. This data may be transmitted to the audiodevice 40 via I/O ports 52 or the communication network 54. In a similarmanner as described above, the audio device 40 may attempt to locallyinterpret the directionality and/or secondary input data. If the audiodevice is unable to interpret the directionality and/or secondary inputdata, the audio device 40 may transmit the directionality and/orsecondary input data to a central processing unit of the industrialcontrol system 11 for further interpretation via communication network54.

The audio device 40 may also include an audio output 56. The audiooutput 56 may be any suitable audio-transmitting component, such as aspeaker. Various status updates and/or error messages may be transmittedfrom the audio output 56. In addition, an operator may interact with theaudio device 40 based at least in part on the audio transmitted from theaudio output 56. In this way, the audio output 56 may facilitatecommunication between an operator and the audio device 40. Furthermore,the operator may leverage the audio output 56 in his/her automationcommands. For example, a person 38 may audibly command the audio device40 to sound an alarm, and in response to determining the associatedautomation command, the audio device 40 may use a control action tooperate itself to emit an alarm sound via the audio output 56.

In some embodiments, the audio device 40 may use the audio data detectedfrom the array of microphones 42 to perform beamforming or spatialfiltering techniques direct audio output to a particular direction. Thatis, in some embodiments, the audio device 40 may include a number ofaudio outputs 56 (e.g., speakers) disposed around the audio device 40.When responding to an information request or indicating that a commandis being implemented, the audio device 40 may use beamforming techniquesto direct the audio in the direction of the source of the originalaudible command. In this way, the audio device 40 may facilitate a morenatural conversational experience with the user.

FIG. 3 is a block diagram of three audio devices 40 includingindications of audio strengths (e.g., 80 dB, 30 dB, 2 dB, and so forth)as detected by various microphones 42 associated with three audiodevices 40. As described above, the microphones 42 detect ambient audiodata and determine automation commands based on the detected audio data.If the audio device 40 determines the detected audio corresponds to acommand, the audio device 40 may reference one or more indications ofaudio strength to determine a direction of the audio data, or adirectionality to the command. Based on this directionality, the audiodevice 40 may assign a subject (e.g., target machine) to the automationcommand. In this way, the audio device determines what component toperform the action of the automation command partly based on thedirectionality associated with the detected audio data. As depicted, theperson 38 speaks a command and the three audio devices 40 detect theaudio data of the command. In addition, as depicted, the audio devices40 may use a relative strength of the received audio (e.g., the receivedaudio data is stronger or louder to a first subset of microphones 42 ascompared to a second subset of microphones 42) to determine thedirectionality of the command manifested as orientation data. From thedirectionality determination, the audio device 40 may determine a targetof the verbal command from the person 38, and may use the orientationdata to determine which machine to perform the detected automationcommand. In addition, the audio devices 40 may also determine a sourcelocation of the verbal command based on the orientation command of theverbal command and/or based on triangulation schemes in embodiments withthree or more audio devices 40 are used in an industrial setting. Itshould be understood that although the audio devices 40 are shown inmultiples, the audio device 40 may operate as a stand-alone device withone microphone 42, a stand-alone device with multiple microphones 42,with other audio devices 40 each having one microphone 42, with otheraudio devices 40 each having one or more microphones 42, or in anycombination of audio devices 40 with any combination of microphones 42.

After determining the automation command from the detected audio data,the audio device 40 may transmit one or more control signals through thecommunication network 54 to perform, or facilitate performing, therequested automation command. To help illustrate, FIG. 4 is a blockdiagram of the industrial automation system 10 including a machine 60and one or more audio devices 40. The industrial automation system 10may include the machine 60 as a stand-alone object, an object includedin the stations 13, 16, 20, 26, 28 to perform a task of the stations 13,16, 20, 26, 28, or included as any suitable component of the industrialautomation system 10. The machine 60 may include any suitable number ofmechanical, electromechanical, pneumatic, or hydraulic devices oroperating mechanisms to perform some industrial operation. In additionto the operating mechanisms, each machine 60 of the industrialautomation system 10 may include a machine control system 61 responsibleat least in part on communicating control operations and/or commands tothe industrial control system 11, which may control monitoring ofmachines or components of the industrial automation system 10. Thiscommunication may occur via a communication network 54, or the like. Forexample, the machine control system 61, via the communication network54, may receive control signals or commands from the industrial controlsystem 11, may transmit statuses indicative of an operation of themachine 60 to the industrial control system 11, may transmit differentinputs from components of the machine 60 to the industrial controlsystem 11, and the like.

Each machine 60 may include one or more interfaces 62 including one ormore input devices 64, such as pushbuttons. Each machine may alsoinclude a human machine interface (HMI) 66 communicatively coupled tothe communication network to facilitate communication between personsoperating, monitoring, or controlling the machine 60, the machinecontrol system 61, and/or the industrial control system 11. The HMI 66may also include one or more input devices 64, such as a keyboard 68,which may enable the person 38 to provide an electrical input into theHMI 66.

In some embodiments, the machine 60 may include one or more stacklights69 designed to emit light in response to control signals transmittedfrom the machine control system 61. In this way, the machine controlsystem 61 may transmit a first control signal to operate the stacklight69 to emit a first color of light and may transmit a second controlsignal to operate the stacklight 69 to emit a second color of lightand/or to stop emission of the first color of light. For example, thestacklight 69 may response to a control signal by strobing, blinking,emitting a solid color of light, changing a color of light emitted,emitting multiple colors of light, changing a color of light emitted, orany combination thereof. The machine control system 61 may operate thestacklight 69 to visually communicate with a person 38 in viewing-rangeof the stacklight 69, for example, by emitting a first color of lightmay correspond to a first message, and a second color of light maycorrespond to a second message. The stacklight 69 may communicatewarnings or statuses related to the stacklight 69, the machine 60, themachine control system 61, the industrial control system 11, and thelike. The machine 60 may change an operation of one or more stacklights69 as an automation command determined via one or more audio devices 40.For example, detected audio data may indicate, “Level 1, green strobe,bright,” with optionally appended information on the number ofstacklights 69 to configure (e.g., one, two, three, ten, or the like,number of stacklights 69), and thus may be used to control operation ofone or more stacklights 69. In response to receiving the automationcommand determined by the audio device 40, one or more of thestacklights 69 may confirm the automation command in a variety ofmanners. For example, the one or more stacklights 69 may confirm receiptby emitting audio data and/or through strobing a green light at a “Level1” (e.g., a configurable level of emitted light) for a predeterminedduration of time (e.g., thirty seconds, forty-five seconds, a minute, orthe like) corresponding to a receipt confirmation.

The machine 60 may include one or more audio devices 40 within alistening range of the machine. Some machines 60 may have one or moreaudio devices 40 while some may have no audio devices 40, based onspecific arrangements of the industrial automation system 10. The audiodevice 40 may take the form of various embodiments, for example, astand-alone audio device 40A mounted on various locations on the machine60, an audio device 40B associated with the stacklight 69, a portableaudio device 40C associated with the person 38 (e.g., a wearable audiodevice 40C), an audio device 40D associated with the interfaces 62, astand-alone audio device 40E located in proximity to the machine 60 butnot directly attached to the machine 60, and the like.

Although the person 38 may operate the machine 60 via the input devices64, such as pushbuttons, keyboards, touch screen inputs, switches, andthe like, the person 38 may also operate the machine 60 via verbal oraudible command to the audio devices 40, as described above. Uponreceiving the verbal command, each audio device 40 may interpret theverbal command, either locally or via communication to the machinecontrol system 61 through the communication network, to determine anaction to perform in response to the verbal or audible command. Forexample, the person 38 may instruct the machine 60 to stop by pressingthe pushbutton or by speaking a verbal or audible command to the audiodevice 40 that requests the stop. In some embodiments, an operator maybe expected to speak an automation command in addition to actuating apush button for additional verification of the automation command. Thismay help to improve robustness, reliability, and accuracy of using theaudio device 40 to operate various machines 60. In response to receivingthe verbal or audible command, the audio device 40 may transmit acontrol signal to the machine control system 61 to cause the machine 60to stop.

As another example, a person 38 may operate a stacklight 69 usingaudible commands directed at the audio device 40. The person 38 mayspeak a command, such as “Light One Red, Strobe,” and the audio device40, in response to determining the corresponding automation command, mayrespond by operating the stacklight 69 to strobe a red colored light.Other commands may include “Light One Yellow, Blink,” “Light ThreeGreen, Steady,” or the like. In this way, different operational statesof the stacklight 69 may be programmed (or any other suitable machine),including a normal operation state and an abnormal operation state(e.g., strobing red light for abnormal operation and steady green lightfor normal operation). It is noted that although light characteristicsare described above, any controllable output characteristic of thestacklight 69, or other suitable machine, may be controlled, programmed,or adjusted via audible command of the person 38. For example, soundoutput parameters, such as intensity, volume, or tone emitted from thestacklight 69 may be changed via audible command directed at the audiodevice 40. Furthermore, in some embodiments, a change made to onestacklight 69 may be applied to multiple stacklights 69 (e.g., a commandto a first stacklight 69 to change a first light color to red causes acommand to be sent to a second stacklight 69 to change a second lightcolor to red).

To help explain the cooperation between the audio devices 40, themachine control systems 61, and the industrial control system 11, FIG. 5is a block diagram of the example communication network 54 that mayfacilitate operation of the industrial automation system 10. Thecommunication network 54 may include multiple machine control systems 61where at least one machine control system 61 is associated with eachmachine 60 of the industrial automation system 10. Each machine controlsystem 61 may include a chassis 80, a power supply 82, a programmablelogic controller (PLC) 84, an industrial system network interface device86 communicatively coupled to the communication network 54, and/orvarious types of input/output (I/O) modules 87. Additionally, someembodiments may include an industrial safety controller 88 andassociated safety I/O devices 90.

Each machine control system 61, HMI 66, and audio device 40 may becommunicatively coupled to the industrial control system 11 via a wiredor wireless connection. In addition, one or more industrial servers 92may be communicatively coupled to the industrial control system 11 tofacilitate authentication of persons 38 operating machines 60 of theindustrial automation system 10 in addition to providing control and/ordatalogging services to a database 94.

During operation, the industrial control system 11 and/or the machinecontrol system 61 may receive inputs from a variety of devices, such asinput devices 64, sensors 31, audio devices 40, and the like. Theseinputs may be status updates, commands, audio data, or control signalsfrom the variety of devices. Based at least in part on the receivedinputs, the industrial control system 11 and/or the machine controlsystem 61 may operate to perform an action, such as changing anoperation of a machine 60, updating an indication on thedisplay/operator interface 32, transmitting a control signal to initiateoperation of a stacklight 69, communicating with the server 92 to storeindication of the input into the database 94, and the like. In addition,the person 38 may use the inputs to control or monitor operatingconditions or operation of the industrial automation system 10. Inresponse to the operating conditions or operation of the industrialautomation system 10, the person 38 may speak an audible command to oneor more of the audio devices 40 to cause the execution of an automationcommand to facilitate in controlling or monitoring components oroperations of the industrial automation system 10.

Keeping the forgoing in mind, FIG. 6 is a flowchart of a method 140 forimplementing automation commands within the industrial automation system10. Although the method 140 is described below as performed by the audiodevice 40, it should be noted that the method 140 may be performed byany suitable processor to perform automation commands based on detectedaudio data. Moreover, although the following description of the method140 is described in a particular order, it should be noted that themethod 140 may be performed in any suitable order.

At block 142, the audio device 40 may receive audio data correspondingto a verbal command from a person 38 from one or more microphones 42.The processor 48 may store the audio data (e.g., digitized data fromanalog input) in the memory 50. In addition, the audio device 40 mayfilter, pre-process, or prepare the received audio data prior tointerpretation to improve interpretability of the received audio data.In other words, the audio device 40 may reduce noise components of thereceived audio data. The audio device 40 may filter received audio data,for example, by using fast Fourier transform (FFT) filtering techniquesto separate the received audio data into its frequency components anddiscard the frequency components corresponding to background noise ofthe industrial automation system 10, however, the audio device 40 mayuse a variety of noise filtering techniques to improve interpretabilityof the received audio data. For example, the audio device 40 may use theFFT filtering techniques, learning algorithms to learn normal orexpected operational noises of the industrial automation system 10 (andthus may ignore the audio data that is expected), neural net filteringtechniques, and the like.

In some embodiments, the audio device 40 may receive certain automationcommands without using a trigger or prompting word (e.g., “hey audiodevice”) that prompts the audio device 40 to use audio data receivedafter the trigger word to determine to the automation command. Instead,the audio device 40 may use repetition of commands (e.g., recited wordor phrase) in place of the trigger word and a directionality of theverbal command is sufficient to cause the audio device 40 to control themachine 60 (assuming the operator is orientated toward the machine 60).For example, the audible command may be stated as, “stop stop stop,” ina direction towards a certain machine. In this way, the first “stop” maybe used to initialize the audio device 40, the second “stop” may be usedto validate or confirm that the command used to initialize the audiodevice is indeed the command that the person 36 intends to perform, andthe third “stop” may be used to initiate the implementation of thecommand. In this way, the third “stop” acts as a trigger or indicationto cause the audio device 40 to implement the detected automationcommand. A particular benefit from using the command without triggerwords is that less memory is used to store the commands because thetrigger words do not have to be stored (e.g., two words are stored forrepetition commanding versus the five words stored for trigger wordcommanding). Another benefit is that the redundant command (e.g., “stopstop stop”) may reduce a likelihood that causal conversation isaccidentally detected by the audio device 40 and used to enact anautomation command. In addition, a confidence level in the detectedautomation command may improve since the audio device 40 uses repeatedinstances of the audio data to interpret the automation commands.

At block 144, the audio device 40 may determine a direction of the audiosource. The audio device 40 may determine a location, orientation, ordirection of the audio source to facilitate determining the automationcommand. In some embodiments, the audio device 40 may communicate withone or more additional audio devices 40 to determine directionality ofthe audio data based at least in part on relative signal strengths ofthe received audio data. After the audio device 40 determines thedirection of the audio data (e.g., where the audio source is locatedrelative to the receiving audio device 40), the audio device 40 mayconvert the directionality into data or information indicative of thelocation of the audio source in the industrial automation system 10(e.g., relative to nearby machines or a direction and/or component theperson 38 aimed, or was speaking in, while verbally commanding the audiodevice), a direction in which the audio data is directed towards, andthe like.

At block 146, the audio device 40 may determine a possible automationcommand for the detected audio data. The audio device 40 may determinethe possible automation command if the detected audio data passes anauthentication process or analysis to verify an operator speaking thecommand. This authentication process may facilitate to increasereliability and accuracy of commanding the audio device 40 to operatemachines 60. In one embodiment, the audio device 40 may determine thepossible automation command through the processor 48 comparing thedetected audio data to a library of audio automation commands, whereeach audio automation command in the library corresponds to one or morecharacteristics used to match the detected audio data to the possibleautomation commands. The library of audio automation commands may bestored in the memory 50 of the audio device 40 or other suitable storagedevice. The audio device 40 may select the automation command that bestmatches the characteristics (e.g., sound wave amplitude) of the detectedaudio data as the possible automation command.

At block 148, the audio device 40 may determine confidence values forthe possible automation command determined at block 146 based onsecondary input (e.g., secondary data received as an input), ifavailable. As mentioned above, the audio device 40 via the processor 48may determine the confidence value based on a variety of secondaryinputs including historical data, expected positions, directions, ororientations of the person 38, inputs from instrumentation, repetitionof commands, and the like. Generally, the secondary input may includeany input or signal received or generated by the machine control system61, the industrial control system 11, or the audio device 40 that may beused to supplement or confirm the automation command. These secondaryinputs provide data related to a presence of a user in the industrialautomation system 10 to the audio device 40 to enhance automationcommand, and subsequent control action, determination. As such, theaudio device 40 may verify the validity of the interpreted automationcommand based on whether an expected secondary input has been receivedby the machine control system 61, the industrial control system 11, orthe audio device 40. In addition, the audio device 40 may receive asecondary input indicative of a measurement and may change an adjustmentto be performed via the automation command based on the secondary input.The secondary input may be any suitable input to help to protect againstthe audio device 40 misinterpreting detected audio data or to helpenhance performing of an automation command. In addition, the secondaryinput may be part of a redundant operation scheme (e.g., n-factorauthentication scheme). For example, an operator may be expected tospeak an automation command in addition to actuating a push button foradditional verification of the automation command. In some embodiments,this n-factor authentication scheme may be performed by more than oneoperator to increase redundancy and reliability of the automationcommand. The n-factor authentication scheme may use any number ofsecondary inputs to verify the audio data of the automation command.

For example, the secondary input may include an input from an inputdevice 64, operated by the person 38, such as a manual switch or button.The secondary input may also include a weight detected on asafety/pressure mat, a signal from a light curtain indicating whetherthe light curtain has been broken or interrupted, an indication from anarea scanner, or inputs from enable pendants, safety gates, guardinterlocks, emergency stop switches, one or more digital cameras, videocameras, infrared sensors, optical sensors, radio frequency energydetectors, sound sensors, vibration sensors, magnetic sensors, heatsensors, pressure sensors, sonar sensors, range imaging technology, andthe like. In one embodiment, the secondary input may contribute todetermining the location and/or orientation of the person 38. Forexample, if the audio device 40 determines that the audible command ofthe person 38 corresponds to an automation command that engages a pressmachine, the audio device 40 may determine whether the detected locationof the person 38 is a safe distance away from the press machine oroutside of an area in which a material may be pressed based on thesecondary input (e.g., pressure mat, image data) before performing theautomation command. As a second example, a person may verbally requestinformation on a machine 60 (e.g., “what is the power output of thismotor?”) and, based on the secondary input (e.g., pressure padindicating where the person 38 is standing, inputs from other audiodevices 40 for triangulation), the audio device 40 may change thepossible automation command from being an automation command directed atthe machine 60 to being a second automation command directed at a secondmachine 60.

In certain embodiments, the audio device 40 may then determine, at block150, whether the confidence value determined at block 148 is greaterthan some threshold. If the confidence value determined at block 148 isgreater than the threshold, the audio device 40 may proceed to block 152and implement the automation command determined at block 146. Inaddition to or in lieu of implementing the automation command, the audiodevice 40 may send a notification to an administrator or an operator ofthe industrial automation system 10 indicating that the automationcommand is being implemented.

For example, if the audio device 40 interprets a verbal command from theperson 38 to correspond to an automation command for changing operationof a machine 60 but the specific machine 60 is not recited in the verbalcommand, the audio device 40 may, at block 152, implement the automationcommand to the particular machine 60 within the closest proximity to theperson 38 or the particular machine 60 that the person 38 is orientatedtoward, as detected by the audio device 40. In some embodiments, theaudio device 40 may determine the orientation of the person 38 based ontriangulation schemes of multiple audio devices 40, signal strength ofdetected audio data, combination of audio data, and the like.

Referring back to block 150, if the confidence value determined at block148 is less than the threshold, the audio device 40 may proceed to block154 and implement a default automation command. The default automationcommand may be a safe state (e.g., power-off, slow down, changeoperation) command for a respective machine control system 61 orrespective machine 60. In one embodiment, the default automation commandmay include continuing the operation of the respective machine controlsystem 61 without adjusting any operations of the industrial automationsystem 10. In another embodiment, at block 154, the audio device 40 maysend a notification requesting a manual confirmation of the automationcommand determined at block 146 or a notification indicating that theconfidence level is low for the respective automation command. Bydetermining whether to implement the recognized automation command basedon the confidence value, the audio device 40 may be safeguarded orprotected against interpreting random audio data as an automationcommand.

In addition to determining automation commands based on detected audiodata and, in some embodiments, secondary inputs, the audio device 40 mayalso analyze ambient audio data to determine if a machine 60 functioningas expected. That is, the audio device 40 may detect abnormal orunexpected audio data and may determine a machinery issue and/orrecommend an action for an operator or administrator to perform tocorrect the machinery issue based on the detected abnormal or unexpectedaudio data. Additional detail with regard to using audio data todetermine machinery issues is provided below with reference to FIG. 7.

Referring now to FIG. 7, FIG. 7 is a flowchart of a method 170 formonitoring a machine 60 of the industrial automation system 10 andperforming automatic machine diagnostics based on detected audio data bythe audio sensor 56. Although the method 170 is described below asperformed by the audio device 40, it should be noted that the method 170may be performed by any suitable processor to perform automationcommands based on detected audio data. Moreover, although the followingdescription of the method 170 is described in a particular order, itshould be noted that the method 170 may be performed in any suitableorder.

At block 172, the audio device 40 may receive detected audio data of themachine 60 or a collection of machines 60 within a proximity of theaudio device 40. Although the detected audio data originates from themachine 60, the audio device 40 receives the detected audio data in asimilar manner as described above with reference to block 142 of FIG. 6.This detected audio data may include ambient noise representative of theoperation of the industrial automation system.

Upon receiving the detected audio data at block 172, the audio device 40may compare the detected audio data to an expected noise profile atblock 174. The expected noise profile may reflect noises correspondingto normal or expected operating conditions of the machine 60 and/or thespace surrounding the machine 60 (e.g., ambient industrial audio). Theexpected noise profile may include actual audio data of the normaloperation, or may include data associated with characteristics of theactual audio data, for example, frequency, amplitude, intensity, orother analysis of the actual audio data. In addition, the noise profilemay include a pattern of sound waves, vibration patterns, and the like.The audio device 40 may compare the detected audio data to the expectednoise profile based on the expected noise profile characteristics. Forexample, the audio device 40 may compare detected audio data to theexpected noise profile based at least in part on frequencycharacteristics. In addition, there may be a directionality associatedwith a given expected noise profile. In this way, the comparison betweenthe detected audio data and the expected noise profile may includecomparing an expected directionality of the audio data to a determineddirectionality of the audio data. This may facilitate identification ofwhen a person 38 is not in a suitable location, or orientation forinstructing the audio device 40 to perform an automation command to aparticular machine.

At block 176, the audio device 40 may determine if the detected audiodata received at block 172 is different from the expected noise profilebased on the comparison between the detected audio data and the expectednoise profile at block 174. The audio device 40 may use a threshold todetermine if the detected audio data and the expected noise profile aredifferent enough to correlate to a machinery issue. For example, if thedetected audio data deviates from the expected noise profile, to thepoint where the deviation exceeds a threshold, the audio device 40 maydetermine that the detected audio data is different from the expectednoise profile.

If the audio device 40 determines the detected audio data is notdifferent from the expected noise profile, the audio device 40, at block172, may receive additional detected audio data to continue to monitoroperation of the machine 60. The detected audio data not substantiallydeviating from the expected noise profile may indicate that the machine60 is operating as expected, or that the machine 60 is not emittingabnormal or unexpected noises, beyond a tolerable threshold.

However, if the audio device 40 determines that the detected audio datais different from the expected noise profile, the audio device 40, atblock 178, may determine that a machinery issue is present based on thedetected audio data received at block 172. The detected audio data maydeviate from the expected noise profile in such a way to suggest orcorrelate to a machinery issue of the machine 60. In addition, in someembodiments, the audio device 40 also analyzes secondary input (e.g.,data associated with a voltage or current spike event, power output orgeneration data, data associated with a power surge event) when thenoise profile deviation occurs. In this way, the audio device 40 maydiagnose machinery issues (e.g., identify a probable cause) of themachine 60 based on the detected audio data and/or the secondary input.In some embodiments, an amount of derivation or a matching of thedetected audio data to the expected noise profile may indicate a type ofdefect, and in some embodiments an amount of derivation between adetermined directionality and the expected directionality indicate atype of defect or help to diagnose the type of defect. For example, ifthe amount of deviation of the detected audio data from the expectednoise profile exceeds a threshold, the audio device 40 may determine acertain defect, however, if the amount of deviation exceeds a secondthreshold, the audio device 40 may determine a different defect.

In addition, the audio device 40 may utilize ranges in diagnosingdefects. For example, a deviation in a first range may indicate a firstdefect while a deviation in a second range may indicate a second defect.Furthermore, in some embodiments, these thresholds or ranges may be usedto determine severity of defect as well, for example, a first amount ofdeviation indicates a defect, and a second amount of deviation indicatesa more severe version of the defect. Thus, the audio device 40 mayanalyze the detected audio data along with secondary data (e.g.,secondary inputs) acquired via various sensors 31 or data sources todetermine a probable cause to explain the deviation from the expectednoise profile (e.g., identify what is malfunctioning with the machine60). Similar to how the audio device 40 at block 146 of FIG. 5determines the automation command based at least in part on the detectedaudio data, the audio device 40, at block 178, may reference noiseprofiles or audio datasets stored in the library to match the detectedaudio data to an expected machinery issue. For example, the machine 60operating with a faulty gearbox may generate a first noise profile, themachine 60 operating with a disconnected wired connection may generate asecond noise profile, and the machine 60 operating as expected maygenerate the expected noise profile, thus the audio device 40 maydetermine machinery issues (e.g., faulty gearbox, disconnected wiredconnection) based at least in part on comparing the detected audio datato these noise profiles or corresponding audio datasets stored in thelibrary.

In some embodiments, the expected noise profile for an industrialenvironment may be associated with a directionality of the ambientnoise. That is, the expected noise profile may include a collection ofsound waveforms associated with an array of microphones 42, as depictedin FIG. 2. In addition, the expected noise profile may include acollection of sound waveforms associated with different audio devices 40in the industrial environment. In any case, as the directionality of thenoise changes, the audio device may detect that the audio data isdifferent from the expected noise profile. In some embodiments, thechange in the directionality may assist the audio device 40 to diagnosea problem or a source of a problem in the industrial environment.

After the audio device 40 detects the machinery issue, the audio device40 may automatically perform a control operation based on the machineryissue, or recommend that a control operation be performed to an operatoror administrator of the machine 60, the industrial control system 11,the industrial automation system 10, and the like. For example, if aparticular machinery issue causes a tank to overfill, the audio device40 may automatically perform, or automatically prompt (e.g., via anotification on the HMI 66) an operator or administrator to perform, acorrective action to prevent the tank overfilling from occurring.

The audio device 40 may also distinguish between human and objects inthe industrial automation system 10 and may implement various controlsand/or cause notification actions based on if a person 38 is presentwithin the industrial automation system 10. For example, the audiodevice 40 may control how lights and/or air conditioning within theindustrial automation system 10 operate based at least in part onwhether a person 38 is within the space of the industrial automationsystem 10.

Keeping this in mind, FIG. 8 is a flowchart of a method 190 forcontrolling one or more devices within the industrial automation system10 based on whether a person 38 or an object (e.g., non-human) arepresent as detected by the audio device 40. Although the method 190 isdescribed below as performed by the audio device 40, it should be notedthat the method 190 may be performed by any suitable processor toperform automation commands based on detected audio data. Moreover,although the following description of the method 190 is described in aparticular order, it should be noted that the method 190 may beperformed in any suitable order.

Referring now to FIG. 8, at block 192, the audio device 40 may receivedetected audio data associated with an element of the industrial controlsystem 11. The audio device 40 may receive the detected audio data in asimilar manner as described above in FIG. 6 at block 142 or in FIG. 7 atblock 172. In addition, the audio device 40 may receive orientation dataindicating directionality of the detected audio data in a similar manneras described above in FIG. 6 at block 144.

At block 194, the audio device 40 may determine whether the detectedaudio data includes human audio embedded within the noise profile (e.g.,signal characteristics defining the detected audio data) of the detectedaudio data. In certain embodiments, the audio device 40 may compare thedetected audio data with templates or known data point arrangements ofhuman audio, or may compare the audio data to expected noise profiles ofhumans. If the determined orientations or the noise profile of thedetected audio data correspond to the known arrangements, orientations,or the expected noise profiles (e.g., emitted sounds, decibel ranges,speech patterns) of humans, the audio device 40 may determine that theelement is a person 38. Otherwise, the audio device 40 may classify theelement as an object, or non-human.

After the audio device 40 determines that the element is either a human(e.g., a person 38) or an object, at block 196, the audio device 40 mayimplement a control action and/or a notification action based on whetherthe element is determined to be the person 38 or an object. The controlaction may include operating a device in the industrial automationsystem 10 or controlling various environmental parameters in theindustrial automation system 10. For instance, the audio device 40 maytransmit a command to a machine control system 61 to change operation ofa heating, ventilation, and air conditioning (HVAC) system in responseto the person 38 not being located within the industrial automationsystem 10. As another example, in the event that the audio device 40determines a person 38 is located within a machine 60 based on theparticular audio data and determined orientation, the audio device 40may implement a control action and/or a notification action associatedwith operating the machine 60 in a safe-state or otherwise changingoperation of the machine 60 to protect the person 38.

With the foregoing in mind, in some embodiments, the audio device 40 maymonitor for distress situations by listening for emergency stop commandsand/or abnormal human sounds. In response to detecting the emergencystop command and/or the abnormal human sounds, the audio device 40 mayanalyze operation of various machines 60 to determine if abnormaloperating conditions exist (e.g., machine 60 indicating an electricalfault or above-average load condition), may communicate with other audiodevices 40 to determine the source location of the audio data, and/ormay determine if various signals from a wearable device (e.g., health orvitals monitoring bracelet, helmet) of the person 38 indicate a distressstate (e.g., accelerated heart rate, accelerated breathing rate, emotionof person 38 as indicated by speech characteristics such as speed ofspeech or a comparison of the speech characteristics to typical tones,cadences, or volumes of operators using the audio device, quick and/orabnormal motions from motion detectors on body). In response to thesesecondary inputs, the audio device 40 may determine the automationcommand to perform in response to the emergency stop command, such aswhich machine 60 to adjust operation of and what adjustment to performto correctly address the emergency stop command (e.g., slow down orpower-off the machine 60). In addition, the audio device 40 may usesecondary inputs from other devices such as pressure pads, lightcurtains, or the lack of input from devices like the pressure padsand/or light curtains to detect if a person 38 is located as expected inthe industrial automation system 10. By combining these various inputsmethods, for example, checking for multiple secondary inputs, the audiodevice 40 may improve operation of the industrial automation system 10and/or may increase safety of persons 38 operating the industrialautomation system 10.

In addition, while following the method 140, the method 170, or themethod 190, the audio device 40 may compare the detected audio data tothe automation commands stored in the library. From the comparison, theaudio device 40 may select an automation command that matchescharacteristics (e.g., frequency, amplitude, intensity) of the detectedaudio data. To determine the possible automation command from theselected automation commands, the audio device 40 may use one or moreadditional inputs, such as the orientation data, or the secondary input,to select or narrow down the possible automation command to the mostlikely automation command to have been provided based on the variouscharacteristics and/or context provided from the additional input.

In some embodiments, the audio device 40 may have access to maintenancerecords and/or data logs associated with the industrial automationsystem 10, the respective machine 60, or the like via the communicationnetwork 54 or direct communication. These records may be stored in thedatabase 94, on the audio device 40, the respective machine 60, or thelike and may enable the audio device 40 to answer queries related tomaintenance records and/or history of the machine 60. For example,questions directed to the audio device 40 may include when the last timea particular machine 60 was serviced, when the last date of refilling ofa vessel was, and the like. Thus, through verbal commands spoken to theaudio device 40, operators may access these maintenance records, datalogs, historic records, or any suitable data stored in memory 50 oraccessible by the communication network 54. After detecting the requestfor information, the audio device 40 may determine a directionality ofthe request, as described above, and query the respective machine 60,the memory 50, some database, or the like for the requested informationbased on the machine 60 or component that the verbal request is directedtowards.

In some embodiments, the audio device 40 indicates its listening to theverbal command while detecting the audio data. While the audio device 40detects the audio data, the audio device 40 may indicate that it islistening to the automation command. The audio device 40 may communicatethe reception of the audio data by changing a color of light emitted byan indicator (e.g., a light-emitting diode indicator), by emitting anaudible indicator (e.g., an audible “beep” or other suitable sound), bychanging a solid light to a flashing light emitted by the indicator, byusing haptic feedback such as if the audio device 40 is included in awearable device (e.g., watch, helmet, button), or any combinationthereof. Furthermore, the audio device 40 may communicate a variety ofstatuses through changing a variety of lights and/or emitting a varietyof audible sounds. For example, in addition to communicating thereception of audio data, the audio device 40 may communicate receptionof an abnormal noise, local processing of the detected audio data,central processing of the detected audio data, a processing error, anerror in the reception of audio data, such as from the surroundingenvironment being too noisy to successfully interpret the command, andthe like.

In some embodiments, the audio device 40 may use voice signatureprocessing to identify, authenticate, and/or authorize a person 38before processing the audio data corresponding to the verbal command. Byanalyzing amplitude of the voice of the person 38, speed of speaking,and/or a frequency of the voice, the audio device 40 may be able todetermine if the person 38 is authorized to use the audio device 40 tooperate one or more machines 60 of the industrial automation system 10.In addition, some audio devices 40 may permit different levels ofauthorization, where a first subset of persons 38 may validly command afirst subset of automation commands, while a second subset of thepersons 38 may validly command a second subset of automation commands.For example, any person 38 may have authorization to verbally command anemergency stop but just certain persons 38 may have authorization toincrease production rates for a particular portion of the industrialautomation system 10.

As described above, the audio device 40 may be able to have naturallanguage conversation with operators and/or administrators, such as theperson 38. For example, the person 38 may ask the audio device 40 aquestion and the audio device 40 may be able to interpret the question,determine the answer to the question based on the context of theconversation or the environment (e.g., facilitated by secondary inputsto the audio devices 40), and communication the answer to the questionback to the person 38 via emission of audio data via the audio output122. This natural language feature may be managed by natural languagesoftware or function that relies on communication algorithms eitherstored locally on the audio device 40 or stored in a memory 50 locationaccessible by the audio device 40. For example, the natural languagesoftware or function may be operated by computing devices associatedwith the server 92 and partially stored in the database 94. In addition,the audio device 40 may converse with the person 38 by emitting audiodata corresponding to the answer through an audio output, such as aspeaker of the audio device 40, so that the person 38 may hear theanswer. It should be understood that any suitable means or technique ofverbal communication may be used by or interpreted by the audio deviceincluding languages such as English, German, Spanish, French, Chinese,or the like.

In some embodiments, the industrial automation system 10 may include arange sensor system to detect positions and movement. In someembodiments, the detected image data may be used by the audio device 40to determine confidence levels for detected audible commands, asdescribed above in block 148 of the method 140. By way of example, therange sensor system of the audio device 40 may include digital cameras,video cameras, infrared sensors, optical sensors (e.g., video/camera),radio frequency energy detectors, sound sensors, sonar sensors,vibration sensors, magnetic sensors, and the like to detect thepositions and/or movements of any element in its viewing window. Thepositions and/or movements may be similarly compared to stored expectedpositions and/or movements, and used to control various components ofthe industrial automation system 10 in addition to detected audio data.In this way, a device or object a person 38 hold or gestures with mayalso be detected, and the positions and/or movements of the object heldby the person 38 may be used to determine an orientation of the person38 and/or a target machine 60 of the verbal command. The detecteddirection and/or commands may be used to determine a confidence valuefor the detected audible command. Example implementations of determiningan automation command based on detected positions or movements of aperson and corresponding confidence levels are described in more detailin U.S. Pat. No. 9,498,885 filed on Nov. 22, 2016, the contents of whichare hereby incorporated by reference in their entirety for all purposes.

With the foregoing in mind, the audio device 40 may be used to programor configure more than one machine 60. In a similar manner, more thanone audio devices 40 may also be configured at the same time with oneautomation command. For example, an operator may command all ten lights(e.g., stacklights 69) to operate to emit green light (e.g., such as astrobing green light). These ten lights may respond with a blinkinglight to communicate to the operator (e.g., verify) that the command wasreceived. The automation command may be used by the audio device 40 tocause control signals to be transmitted to each of the ten lights for asubstantially simultaneous or a sequential light operation adjustment(e.g., to change an operation of one or more lights at a time). Someembodiments may use a second verification action to communicate to theoperator that the command was received and enacted.

Technical effects of the present disclosure include techniques foroperating an industrial automation system through verbal commands andadditional inputs. An audio device designed for industrial applicationsmay detect audio data. Based upon the detected audio data, the audiodevice may determine one or more possible automation commands. In someembodiments, the audio device implements an automation command to adjustoperation of a component of the industrial automation system, topower-down or power-on a component of the industrial automation system,to answer a question received as a verbal command, and the like. In thisway, the audio device may facilitate handless or remote operation ofvarious devices of the industrial automation system by an operatorwithin line of sight of the various devices, thereby increasing safetyand improving methods of managing industrial automation systems.

While only certain features of the disclosure have been illustrated anddescribed herein, many modifications and changes will occur to thoseskilled in the art. It is, therefore, to be understood that the appendedclaims are intended to cover all such modifications and changes as fallwithin the true spirit of the disclosure.

The invention claimed is:
 1. A method, comprising: detecting, via anaudio device, audio data associated with an object corresponding to anon-human in an industrial environment; determining, via the audiodevice, orientation data configured to indicate a directionality of theobject based at least in part on a strength of the audio data;determining, via the audio device, that the audio data is different froman expected noise profile of the object; determining, via the audiodevice, an automation command that corresponds to a control actionconfigured to control one or more operations of a component in anindustrial automation system in response to determining that the audiodata is different from the expected noise profile of the object andbased on the orientation data; and sending, via the audio device, theautomation command to the component, wherein the automation command isconfigured to cause the component to implement the control action. 2.The method of claim 1, wherein determining the automation commandcomprises determining, via the audio device, whether one or moreabnormal operating conditions are present in the component based on theaudio data and the orientation data.
 3. The method of claim 2, whereinthe one or more abnormal operating conditions correspond to a faultpresent in the component, the component operating at a load conditionthat exceeds a threshold, or both.
 4. The method of claim 1, comprising:determining, via the audio device, a location of the object based on theorientation data; and determining, via the audio device, the automationcommand based on the location of the object.
 5. The method of claim 1,comprising determining, via the audio device, that the audio data isdifferent from the expected noise profile of the object at least in partby comparing the audio data with a plurality of noise profiles of theobject stored in a database.
 6. The method of claim 1, comprisingdetermining, via the audio device, the automation command based on theaudio data and the orientation data being indicative of the object beinglocated within a proximity distance to the component.
 7. A device,comprising: a memory comprising one or more instructions; and aprocessor configured to execute the one or more instructions, whichcause the processor to perform operations comprising: detecting audiodata associated with an object corresponding to a non-human from one ormore audio sensors; determining orientation data configured to indicatea directionality of the object with respect to the device based at leastin part on a strength of the audio data associated with the object;determining that the audio data is different from an expected noiseprofile of the object; determining an automation command thatcorresponds to a control action configured to control one or moreoperations of a machine in an industrial automation system in responseto determining that the audio data is different from the expected noiseprofile of the object and based on the orientation data; and sending theautomation command to the machine, wherein the automation command isconfigured to cause the machine to implement the control action, whereinthe control action is configured to cause the machine to adjust the oneor more operations.
 8. The device of claim 7, wherein the processor isconfigured to perform the operations comprising determining that theaudio data is different from the expected noise profile of the object atleast in part by comparing the audio data to a plurality of noiseprofiles of the object.
 9. The device of claim 7, wherein the processoris configured to perform the operations comprising sending anotification to a computing device in response to determining that theaudio data is different from the expected noise profile of the objectand the orientation data.
 10. The device of claim 7, wherein the machineis configured to control one or more environmental parameters.
 11. Thedevice of claim 10, wherein the machine comprises a heating,ventilation, and air conditioning (HVAC) system.
 12. The device of claim7, comprising a plurality of sensors configured to receive temperaturedata, pressure data, occupancy data, or any combination thereof, whereinthe processor is configured to perform the operations comprisingdetermining the automation command based on the temperature data, thepressure data, the occupancy data, or any combination thereof.
 13. Thedevice of claim 7, wherein the processor is configured to perform theoperations comprising identifying the machine associated with theautomation command based on the orientation data.
 14. The device ofclaim 7, wherein the processor is configured to perform the operationscomprising determining one or more abnormal operating conditions in themachine based on determining that the audio data is different from theexpected noise profile of the object.
 15. A tangible, non-transitorycomputer-readable medium configured to store instructions executable bya processor of an electronic device that, when executed by theprocessor, cause the processor to perform operations comprising:detecting audio data associated with an object corresponding to anon-human from one or more audio sensors; determining orientation dataconfigured to indicate a directionality of the object with respect tothe one or more audio sensors based at least in part on a strength ofthe audio data; determining that the audio data is different from anexpected noise profile of the object; determining an automation commandthat corresponds to a control action configured to control one or moreoperations of a machine in an industrial automation system in responseto determining that the audio data is different from the expected noiseprofile of the object and based on the orientation data; and sending theautomation command to the machine, wherein the automation command isconfigured to cause the machine to implement the control action, whereinthe control action is configured to cause the machine to adjust the oneor more operations.
 16. The non-transitory computer-readable medium ofclaim 15, wherein the instructions, when executed by the processor,cause the processor to perform the operations comprising determiningthat the audio data is different from the expected noise profile of theobject at least in part by comparing the audio data with a plurality ofnoise profiles of the object stored in the non-transitorycomputer-readable medium.
 17. The non-transitory computer-readablemedium of claim 15, comprising instructions, when executed by theprocessor, cause the processor to perform the operations comprising:receiving additional audio data from an additional audio device;determining a location of the object based on one or more triangulationschemes and the additional audio data; and determining the automationcommand based on the location of the object.
 18. The non-transitorycomputer-readable medium of claim 15, comprising instructions, whenexecuted by the processor, cause the processor to perform the operationscomprising: comparing the audio data with a second set of expected dataprofiles; determining a state of the object based on comparing the audiodata with the second set of expected data profiles; and determining theautomation command based on the state of the object.
 19. Thenon-transitory computer-readable medium of claim 15, comprisinginstructions, when executed by the processor, cause the processor toperform the operations comprising determining one or more abnormaloperating conditions in the machine based on determining that the audiodata is different from the expected noise profile of the object.
 20. Thenon-transitory computer-readable medium of claim 19, wherein the one ormore abnormal operating conditions comprise an electrical fault in themachine, a load condition associated with the machine that exceeds anaverage load condition, or both.